An inference algorithm is any algorithm that takes input observations from the world and makes inferences about the causes of those observations. Different types of inference algorithms are stereo algorithms and computer vision algorithms. Inference algorithms are used in several areas including three-dimensional imaging and voice recognition and natural language understanding. As technology utilizing inference algorithms advances, methods for determining the accuracy of these algorithms is essential. Accuracy determinations are based on statistical characterizations of the performance of the algorithm.
One possible statistical characterization of the performance of an inference algorithm is in terms of the error of the matches compared to “ground truth.” Comparison with ground truth requires the build up a corpus of ground truth observations and training the algorithm based on the corpus. If sufficient quantities of ground truth are available estimating the distribution of errors over many image pairs of many scenes is relatively straight forward. This distribution could then be used as a prediction of the accuracy of matches in new images.
The comparison to ground truth, however, has several drawbacks. Acquiring ground truth for any scene is an expensive and problematic proposition, as several observations must be recorded. Also, aquiring ground truth for any scene is extremely time and labor intensive. Furthermore, any ground truth measurements are subject to discrepancies in the measurements, therefore reducing the accuracy of the performance.
Accordingly, a need exists for a method for measuring the accuracy of an inference algorithm that does not require comparison to ground truth. A need exists for a method that can satisfy the above need and that is not time and labor intensive. Furthermore, a need exists for a method that can satisfy the above needs and that is cost effective and not overly expensive.